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EVALUA  TING  MISSION  TRAINING  FIDELITY  REQUIREMENTS 

Due  to  infrequent  training  while  deployed,  warfighters’  skills  can  decay,  thus  creating  a  training  gap. 
Deployed  training  has  historically  been  hindered  by  restrictions  to  live-fly  training  opportunities  due  to 
factors  such  as  ops-tempo,  airspace/range  restrictions,  security  issues,  alert  requirements,  and  wartime  rules 
of  engagement.  In  order  to  maintain  high  proficiency  and  readiness  levels,  changes  are  needed  in  standard 
training  programs  while  warfighters  are  deployed.  The  United  States  Air  Force’s  Distributed  Mission 
Operations  (DMO)  concept  has  become  critical  to  warfighter  training  across  all  mission  areas.  Despite  an 
increased  reliance  on  DMO  training,  a  deployable  DMO  training  capability  does  not  exist  to  provide  the 
critical  training  opportunities  previously  unavailable  during  extended  deployments.  Moreover,  while 
researchers  have  discussed  and  described  the  tradespace  associated  with  varying  levels  of  physical  and 
functional  fidelity,  opportunities  to  conduct  controlled  studies  to  examine  and  quantify  “how  much  of  what 
kind  of  fidelity”  for  skills  improvement  and  maintenance  are  few  and  far  between.  This  presentation  will 
describe  a  series  of  research  studies  underway  at  the  Warfighter  Readiness  Research  Division  in  Mesa 
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Arizona  designed  to  assess  within-training  and  training  transfer  to  live  operations  for  both  individual  and 
tactical  teams,  with  varying  levels  of  fidelity  in  the  training  environment.  These  studies  are  examining  a 
variety  of  specific  parameters  associated  with  achieving  and  maintaining  combat  readiness  and  proficiency. 
We  conclude  with  discussion  of  our  longer-term  plan  to  demonstrate  and  quantify  the  impact  of  deployed 
training  and  rehearsal  environment  on  transfer  to  the  operational  environment  and  to  subsequent  retraining 
intervals. 


1.0  CHANGING  TRAINING  NEED 

A  shift  in  available  resources  and  changes  in  the  strategic  environment  demand  a  transformation  in  how  we 
prepare  the  warfighter.  The  financial  cost  of  actual  training  events  has  increased,  while  the  opportunities  to 
conduct  actual  training  events  have  decreased.  Operational  tempo,  security  issues,  alert  requirements, 
environmental  restrictions,  encroachment  on  training  areas,  and  the  reduced  tolerance  of  the  general 
population  for  close  proximity  of  military  training  have  all  combined  to  make  it  more  difficult  to  carry  out  the 
type  of  live  training  activities  common  20  or  even  10  years  ago  [1].  Additionally,  the  emphasis  has  shifted 
towards  smaller  deployed  forces  capable  of  operating  anywhere,  anytime,  day  or  night,  and  in  all  weather. 
Training  must,  therefore,  transform  to  support  the  needs  of  the  force.  Often,  our  warfighters  are  deployed  for 
extended  periods  of  time,  but  without  sufficient  opportunity  to  train/maintain  their  skills.  This  insufficient 
deployed  training  results  in  skill  proficiency  loss,  reduced  readiness,  and  additional  training  time  later  to 
reconstitute  those  skills.  Unfortunately,  deployable  systems  for  use  in-theatre  do  not  currently  exist  to  fill  this 
training  gap.  In  order  to  maintain  high  proficiency  and  readiness  levels,  changes  are  needed  in  standard 
training  programs  while  warfighters  are  deployed. 

Meeting  the  training  need  of  the  deployed  warfighter  encompasses  a  range  of  requirements.  Ideally,  a 
deployable  system  would  allow  the  warfighter  to  train  critical  aspects  of  the  “job”,  i.e.,  the  transfer 
environment.  The  “job”  is,  of  course,  more  than  what  is  learned  in  the  classroom  and  more  than  just 
checklists  and  emergency  procedures.  A  warfighter  must  be  prepared  for  Mission  Essential  Competencies 
(MECs),  which  are  the  higher-order  individual,  team,  and  inter-team  competencies  that  a  fully  prepared  pilot, 
crew,  flight,  operator,  or  team  requires  for  successful  mission  completion  under  adverse  conditions  and  in  a 
non-permissive  environment  [2].  The  conditions  of  performance  in  this  MEC  definition  are  most  important. 
The  focus  here  is  on  combat  conditions  to  define  standards  of  warfighter  performance  [3]. 

1.1  Distributed  Mission  Operations  Training  as  a  Solution 

Distributed  Mission  Operations  (DMO)  training  capabilities  can  be  generally  defined  as  affording  the  ability 
to  bring  a  number  of  warfighters  together  to  train  complex  individual  and/or  team  tasks  during  the  course  of 
larger-scale,  realistic  combat  missions  [4].  The  very  definition  of  DMO  appears  to  align  closely  with  the 
combat-relevant  conditions  of  perfonnance  necessary  as  part  of  maintaining  MEC  proficiency.  Interestingly, 
in  1991,  a  time  period  just  preceding  DMO,  researchers  [5]  surveyed  94  F-15  air  combat  pilots  and  discovered 
higher  order  experiential  areas  as  receiving  less  than  adequate  training  in  their  current  unit.  The  experiences 
identified  were:  bogey,  reaction  to  SAMs,  dissimilar  air  combat  tactics,  all-weather  employment,  electronic 
countermeasure  employment,  communications  jamming,  low  altitude  tactics,  chaff/flare  employment,  escort 
tactics,  track-while-scan  assessment,  and  work  with  the  Air  Weapons  Controller.  The  F-15  pilots  also  felt  all 
of  these  experiences  were  better  suited  to  the  simulator,  this  opinion  provided  even  during  an  era  in  which 
much  of  the  benefits  of  networking  and  the  emerging  DMO  environment  were  still  absent!  The  early 
evidence  for  simulator  advocacy  was  not  isolated.  Other  practitioners  in  1993  [6]  interviewed  99  F-16  pilots, 
who  reported  that  the  highest  “difficult  aspects”  of  attaining/maintaining  mission-ready  status  were  weapons 
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delivery,  radar  interpretation,  electronic  combat,  cockpit  switchology,  and  air-to-air  combat.  When  the  F-16 
pilots  were  asked  which  ground-based  medium  they  would  like  to  see  used  more,  the  most  preferred  option 
was  the  simulator. 

Another  large  sample  pilot  opinion  study  was  conducted  recently,  specifically  on  contemporary  DMO 
training.  In  that  study  [7]  the  authors  surveyed  327  F-16  pilots  and  49  Airborne  Warning  and  Control  Station 
controllers,  who  based  their  opinions  on  experiences  obtained  during  a  five-day  DMO  training  research 
program  containing  over  40  total  air  combat  scenarios.  “I  would  recommend  this  training  experience  to  other 
pilots/controllers”  was  rated  with  the  highest  rating  possible  of  “Strongly  Agree”  by  all  but  one  of  49 
controllers  and  all  but  16  of  327  pilots.  When  asked  to  rate  to  what  extent  the  DMO  environment  provided  the 
45  different  F-16  critical  air-to-air  experiences  (defined  by  the  MEC  process),  the  pilots  rated  that  38 
experiences  (84%)  could  be  obtained  “to  a  moderate  extent”  or  higher,  more  than  all  seven  other  training 
environments  surveyed. 

These  large  sample,  opinion-based  studies  reveal  warfighter  advocacy  for  DMO,  but  how  effective  is  it  as  a 
training  device?  Recently,  a  large-scale  study  was  done  specifically  to  examine  DMOs  training  effectiveness 
[8].  76  F-16  teams  (384  pilots)  flew  for  one  week  in  DMO  and  were  “tested”  at  the  beginning  of  the  week  and 
again  at  the  end  of  the  week  on  mirror-image  point  defence  scenarios.  On  the  post-test,  the  teams,  on  average 
per  scenario,  allowed  58.33%  fewer  enemy  strikers  to  target,  killed  9.20%  more  enemy  aircraft,  reduced  F-16 
mortalities  by  54.77%,  and  registered  55.20%  less  time  allowing  hostiles  into  vulnerable  ranges.  Also  part  of 
this  study,  expert  observer  ratings — both  those  taken  in  real-time  and  those  done  according  to  a  scientifically 
blind  protocol — revealed  statistically  significant  improvements  as  a  function  of  the  DMO  training.  The 
improvements  in  expert  ratings  were  found  both  for  briefs/debriefs  and  also  for  mission  execution, 
corroborating  the  objective  results.  DMO  training  is  obviously  an  effective  training  paradigm,  but  no  DMO 
solution  currently  exists  to  fulfill  the  deployed  warfighter’s  training  need.  So,  what  are  the  issues  involved  in 
researching  and  building  a  deployable  DMO  solution? 

1.2  Issues  in  Researching  and  Building  a  Deployable  Distributed  Mission  Operations 
Tactical  Trainer 

Designers  and  users  of  training  devices  often  attempt  to  replicate  as  many  physical  and  functional  stimuli  as 
possible  in  a  training  device  [9].  The  goal  here  is  often  to  increase  physical  fidelity  (i.e.,  look  &  feel)  and 
functional  fidelity  (i.e.,  the  dynamics,  or  “actions”),  two  arguably  interdependent  system  aspects  [10].  Prior 
research  addresses  familiar  themes  in  building  the  “right”  simulation  training  environment,  with  many  of  the 
issues  revolving  around  the  level  of  fidelity  needed.  In  general,  the  literature  tends  to  support  the  notion  that 
higher  fidelity  will  translate  into  a  better  training  device  (e.g.,  [11]),  but  this  does  not  mean  that  effective 
training  devices  must  all  be  high  fidelity.  Indeed,  even  low  fidelity  pilot  trainers  absent  of  stick  and  throttle 
have  shown  to  positively  transfer  compared  to  a  control  group  [12].  Further,  some  studies  have  shown  that 
added  fidelity  does  not  significantly  contribute  to  performance  [13].  As  a  rule-of-thumb,  however,  increasing 
fidelity  tends  to  increase  the  training  utility  and  value,  perceived  and/or  real.  But,  increases  in  fidelity  are 
coincident  with  costs  [14].  Increasing  costs  almost  inevitably  leads  to  fidelity  compromises.  One  solution  is 
to  build  a  reconfigurable  system  capable  of  training  different  platforms  (e.g.,  [15]).  However,  adopting  a 
reconfiguarable  system  does  not  ameliorate,  and  may  exacerbate,  the  difficulty  of  choices  needed  to  be  made 
when  it  comes  to  cost  versus  fidelity. 

To  build  a  capable  deployable  DMO  tactical  trainer  (in  our  case  an  F-16  tactical  trainer),  a  number  of 
objectives  are  desirable,  including  the  ability:  (a)  to  train  forces  with  a  realistic  mix  of  experiences  from 
strategic  to  tactical  levels;  (b)  to  integrate  live,  virtual,  and  constructive  elements  within  a  common  synthetic 
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environment;  (c)  to  integrate  low  density/high  demand  assets  through  distributed  simulation;  (d)  to  practice 
missions  requiring  a  “total  team”  environment  to  meet  national  objectives;  (e)  to  meet  institutional  training 
goals  and  needs  for  individual,  collective,  and  joint/coalition  training;  (f)  to  tailor  training  for  individuals  and 
teams;  (g)  to  support  Air  Expeditionary  Force  training  and  preparatory  spin-up  and  sustain  wartime  readiness; 
and,  (h)  to  provide  joint  training  with  command  linkages.  To  work  towards  these  objectives,  scores  of 
decisions  must  be  made  regarding  physical  fidelity  (e.g.,  display  field  of  view,  stick  and  throttle,  switches, 
etc.),  and  functional  fidelity  (e.g.,  flight  model,  weapons  models,  threat  generation  system,  etc.).  Of  course, 
the  same  central  question  presents  itself:  In  building  a  DMO  environment,  how  do  we  know  which  fidelity 
components,  when  interconnected,  provide  the  best  training  value  for  the  money  spent?  And,  once  the  system 
is  built,  additional  questions  must  be  addressed,  including  those  that  relate  to  assessing,  tracking,  and 
comparing  warfighter  performance  both  within  a  deployable  DMO  system  and  at  other  locations.  That  is, 
what  approach  needs  to  be  taken  to  assess  combat  readiness  and  proficiency  across  sites?  Such  questions, 
properly  addressed,  would  provide  answers  for  the  degree  of  training  transfer  between  environments,  skill 
acquisition  rates,  and  recommended  DMO  retraining  intervals. 


2.0  APPROACH 

To  build  and  complete  sound  empirical  evaluations  of  all  possible,  practical,  variations  of  deployable  DMO 
tactical  trainers  is  cost-prohibitive,  especially  given  the  rapid  change  in  technology  progress  and  alternatives. 
Therefore,  the  approach  of  choice  should  be  appropriately  aggregated  so  as  to  avoid  assessing  all  possible 
technology  interaction  effects,  yet  still  provide  meaningful  information.  A  simple  approach  yielding  data- 
based  recommendations  is  needed.  Likewise,  an  approach  to  assessment,  tracking,  and  comparing 
perfonnance  needed  to  be  simple  if  it  is  to  be  implemented  successfully  across  sites.  These  common 
assessments  would  provide  the  proficiency-based  data  needed  to  help  answer  questions  regarding  skill 
acquisition,  decay,  training  transfer,  etc. 

2.1  Building  and  Evaluating  the  System 

Our  goal  for  the  deployable  system  is  to  meet  the  needs  of  the  warfighter  with  a  level  of  fidelity  that  best 
prepare  him/her  for  combat  experience.  To  this  end,  we  are  able  to  leverage  outputs  from  the  MEC  process,  in 
which  operational  warfighters  are  brought  together  and  define  a  critical  list  of  experiences  they  believe  would 
fully  prepare  a  warfighter  for  combat.  In  the  case  of  the  F-16,  the  MEC  process  identified  171  total 
experiences  across  a  variety  of  mission  types  (e.g.,  air-to-air,  air-to-ground,  suppression  of  enemy  air 
defense).  Selected  examples  of  the  F-16  experiences  are  provided  in  Table  1.  Note  that  many  of  the 
experiences  require  the  simulation  to  be  a  system  of  technologies  to  include  sophisticated  capabilities  (e.g., 
night  vision,  various  threat  capacities)  and  multi-player  connectivity  (e.g.,  buddy  lasing,  large  force 
employment). 


Table  1:  Selected  Mission  Essential  Competency  Experiences  for  the  F-16 


Performing  buddy  lasing 
Employing  weapons  with  degraded  systems 
Operating  in  mountainous  versus  flat  terrain 
Large  force  employment _ 


Unlocated  or  pop-up  threats 
Using  chaff/flare  for  threat  reaction 
Seeing  surface-to-air  threat  launches  under  NVG 
Degraded  aircraft  avionics _ 


These  171  experiences  provided  us  with  a  comprehensive  set  of  mission/tactical  level  ideals.  To  augment  this 
exhaustive  tactical  level  list,  we  chose  to  add  the  emergency  procedures  (EP)  list  from  the  United  States  Air 
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Force  Dash  One  document  for  the  F-16.  The  MEC  experiences  and  the  EPs,  if  fully  and  faithfully  realized  in 
any  DMO  environment,  would  represent  the  highest  trainability/usability  standards  we  could  hope  to  achieve. 
Flence,  the  warfighter- centered  list  therefore  became  our  system  evaluation  criteria. 


Since  scientifically  evaluating  all  technological  permutations  was  cost-prohibitive  and  impractical,  our  initial 
deployable  DMO  build/configuration  was  decided  upon  by  examining  the  spectrum  of  MEC  experiences  for 
common  underlying  functionalities/requirements.  Examples  of  these  included  F-16  flight  dynamic  model, 
threat  generation  system,  avionic  displays,  visual  display  system,  simulator  connectivity,  weapons  models, 
terrain  database,  etc.  For  each  common  area,  an  initial  low-cost  solution  was  identified,  procured,  and 
integrated,  thus  creating  an  initial,  baseline  deployable  DMO  configuration.  This  configuration  was 
documented  in  a  survey  and  is  in  the  process  of  being  evaluated  by  F-16  Subject  Matter  Experts  (SMEs),  who 
separately  rate  the  ability  of  the  system  to  provide  each  of  the  MEC/EP  experiences.  Analyses  will  reveal  the 
system’s  trainability/usability  by  each  experience  and  EP,  by  aggregated  mission  area  (e.g.,  air-to-ground), 
and  overall.  Examining  commonalities  amongst  the  lowest  rated  MEC/EP  experiences  will  reveal  the 
technological  deficiencies  that  need  to  be  addressed  (e.g.,  insufficient  field  of  view).  The  deficient 
functionalities  will  form  a  follow-up  survey.  In  this  survey  SMEs  will  rate  each  deficiency  against  each  of  the 
MEC  experiences/EPs  for  the  degree  to  which  each  detracts  from  the  ability  to  gain/train  that  experience/EP. 
The  results  will  provide  a  priority  order  of  fidelity/technology  areas  to  be  upgraded,  constrained  by  cost 
considerations. 

Once  this  evaluation/development  cycle  is  complete,  a  number  of  scientific  opportunities  become  available. 
First,  we  can  re-evaluate  the  (upgraded)  system  to  determine  the  training  value  added  for  the  cost  incurred. 
Second,  we  can  implement  the  same  survey  evaluation  process  on  other,  higher  fidelity  environments  for 
additional  comparison  of  trainability/cost.  And  third,  in  conjunction  with  data  from  the  first  two  efforts,  we 
can  re-use  the  same  two-cycle  evaluation/deficiency  survey  process  iteratively  in  building  the  deployable 
DMO  system  to  determine  the  point  of  diminishing  returns,  in  terms  of  added  trainability/usability  for 
additional  costs.  Though  this  approach  cannot  delineate  all  possible  interaction  effects  between  technology 
subsystem  components,  it  does  provide  us  cost-effective  data-driven  assessments  of  the  overall  system  and  the 
fidelity/cost  trade-offs  involved. 

2.2  Need  for  Standardized  Assessment  Suite  (SAS)  and  protocol. 

Once  a  configuration  of  the  deployable  DMO  trainer  is  settled  and  deployed  for  training  use,  a  number  of 
research  questions  arise.  Research  will  need  to  be  undertaken  in  the  deployable  trainer  to  examine  skill 
acquisition  rates  needed  to  meet  proficiency,  skill  decay  rates  (for  new  and  experienced  pilots),  training 
effectiveness,  and  degree  of  transfer  between  the  deployable  trainer  and  live-fly  and  other  DMO  training 
environments.  To  examine  these  issues,  a  central,  critical  need  is  to  quantitatively  assess  proficiency  using  a 
standard,  controlled  protocol.  Doing  so  permits  comparisons  of  warfighter  skill  levels  both  longitudinally  and 
across  sites.  To  meet  this  assessment  criterion  for  addressing  the  research  objectives,  a  standardized 
assessment  suite  (SAS)  will  need  to  be  developed  to  evaluate  warfighter  performance  within  not  only  the 
deployable  trainer,  but  also  any  other  DMO  location  used  as  part  of  the  investigation. 

Conceptually,  the  SAS  approach  provides  a  common  method  to  compare  warfighter  proficiency  across  all 
DMO  environments  (e.g.,  simulator,  live-fly  training,  exercises,  and  potentially  combat  locations) — similar  to 
the  method  of  using  Graduate  Record  Exams  (GRE)  across  universities  to  assess  student  proficiency.  At 
periodic  intervals  within  the  deployable  trainers  (and  other  DMO  locations),  a  set  of  missions  would  be 
performed  under  strict,  standardized  data  collection  protocols.  The  set  of  missions  to  be  performed  would  be 
a  sample  of  the  suite,  representing  a  spectrum  of  difficulty.  The  scenarios  comprising  the  mission  “set”  to  be 
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ORGANIZATION 


periodically  flown  would  be  a  sample  of  the  overall  library  of  available  “test”  missions.  The  assessments  for 
all  scenarios  would  be  standardized,  leveraging  final  product  versions  from  three  projects  currently  under 
development:  (1)  MEC  vignette  knowledge  test  [16],  (2)  subjective  assessment  of  skill  proficiency  [17],  and 
(3)  objective  data  collected  directly  from  the  simulation  systems  [18]. 

At  each  DMO  location,  experiment  control  of  the  standardized  assessment  suite  and  a  protocol  for  managing 
the  training  conditions  and  for  data  collection  will  be  required.  However,  no  restriction  is  needed  regarding 
who  participates  and  when,  only  that  demographic  data  is  captured  on  each  participant  when  he/she  performs 
a  SAS  set  of  scenarios  (e.g.,  dates  of  participation  at  each  location  and  relevant  moderator  training  events  such 
as  weapons  school).  By  utilizing  this  approach,  data  can  be  collected  on  all  participants  through  each  DMO 
environment,  thereby  maximizing  sample  size.  By  controlling  the  SAS  tests  and  tracking  other  moderating 
demographic  variables  of  interest,  sufficient  data  can  be  collected  at  each  site  to  allow  for  the  application  of 
multiple  regression  and/or  mixed  models  of  analysis  to  subsets  of  the  database  that  address  specific  research 
questions  of  interest. 

These  mixed  models  of  analysis  will  use  the  standardized  assessment  suite  as  a  multivariate  outcome  space 
and  use  other  variables  (site/fidelity,  pilot,  important  potential  moderators)  as  predictors  to  address  the 
research  questions  of  skill  acquisition  rates  to  meet  proficiency,  skill  decay  rates,  training  effectiveness, 
fidelity  to  meet  proficiency  criterion,  and  degree  of  transfer  between  DMO  environments.  In  addition  to  the 
benefits  of  collecting  data  observationally  at  each  DMO  location  (i.e.,  not  having  predefined  experimental 
conditions)  and  therefore  maximizing  sample  size,  the  SAS  approach  was  also  chosen  because  it:  (a) 
minimizes  necessary  DMO  environment  controls,  this  method  is  likely  to  introduce  the  lowest  amount  of  risk; 
(b)  requires  fewer  controls  to  implement,  track  and  enforce,  leading  to  the  lowest  research  labor  cost  possible; 
and  (c)  ensures  data  in  the  alternative  environments  will  be  tracked  according  to  their  operational  uses, 
resulting  in  an  ability  to  compare  DMO  training  strategy  implementations  and  relative  effectiveness  of  each 
environment.  This  SAS  approach,  if  successful  within  the  deployable  trainers  and  a  few  other  DMO 
locations,  would  then  become  a  key  component  in  the  USAFs  training  transformation  plan  to  augment  the 
Ready  Aircrew  Program  with  competency-based  assessment. 
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Background 


•  To  keep  pilots’  proficiency  at  a  high  level,  training  must 
transform  due  to: 


—  Increased  costs:  Live-fly  expenses  costly,  absolute  number  of  large  scale 
exercises  relatively  infrequent 

—  Real  world  restrictions:  Operations  tempo,  airspace  restrictions,  security 
concerns,  and  ROE  limit  live-fly  training  capabilities  in  deployed  settings. 

—  Strategic  and  deployment  shifts:  Emphasis  on  smaller  forces  deployed 
anywhere,  anytime,  in  all  weather  and  non-permissive  environments. 


Background 


Distributed  Mission 
entities  fulfills  many  o 


Operations  (DMO),  a  system  of  networked 
"the  training  needs: 


—  Provides  multi-player  teams  opportunities  to  train  on  complex  individual 
and/or  team  skills  in  large  scale,  realistic  combat  missions. 

—  Can  be  linked  to  a  Performance  Evaluation  Tracking  System  that 
transparently  measures  behaviors. 
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DMO  Justification 


•  The  evidence  that  DMO  can  meet  AF  training  needs  is  strong: 


—  1991  Survey  (Just  preceding  DMO):  94  F-15  pilots  reported 
inadequate  training  in  their  current  unit  for  (among  others): 

•  Multi-bogey  engagements 

•  Low-altitude  and  dissimilar  air  combat  tactics 

•  All  weather  conditions 

•  Jamming 

•  Work  with  air  weapons  controllers 

—  1993  Interviews:  99  F-1 6  pilots  reported  training  inadequacies 
with  many  “difficult  aspects” 


—  The  1991  and  1993  pilots  specifically  identified  simulation  systems 
as  the  training  system  most  likely  to  correct  training  shortfalls. 


—  2002-2005  DMO  Study:  76  F-1 6  teams  (N  =  327  pilots), 

participants  in  week-long  DMO  training,  strongly  endorsed  DMO 
for  providing  Mission  Essential  Competency  experiences. 


DMO  Justification 


•  Warfighter  endorsement  is  a  necessary,  but  not  sufficient  condition  to 
establish  DMO  efficacy.  The  ’02-’05  DMO  Study  provides  objective  evidence. 


—  76  teams  (327  pilots)  flew  pre/post  scenarios: 

•  Were  all  point  defense  Bench5A 

•  Were  mirror  images  -  - _ _ 

^  nj  » 

•  Equally  complex  *  > 

I  J  I  Drag  at  30  NM  for 


Bench-5B 


9K 


28K 


»- 

>►- 


Drag  at  30NMfor 
1  min  and  descend  to  1 SK 


vV  24K)^--- 

T —  L... 

K  A 


—  The  performance  changes  from  the  beginning  to  end  of  week  provide 
unequivocal  evidence  of  DMO  training  efficacy,  e.g.: 

•  58.33%  fewer  enemy  strikers  reaching  target, 

•  9.20%  more  enemy  aircraft  killed, 

•  64.77%  reduction  in  F-16  mortality,  and 

•  55.20%  reduction  of  time  spent  in  vulnerable  ranges. 

•  Above  performance  metrics  corroborate  real-time  and  blind  ratings  by 
expert  observers. 


Deployed  Training 


•  “Bathtub  effect” — frequently  just  burning  dinosaurs 

•  A  deployable  form  of  DMO  is  a  significant  step  toward  meeting 
deployed  training  needs  of  the  Air  Force 

•  Conclusion:  Need  affordable,  deployable,  systems  so  that 
significant,  Mission  Essential  Competency  (MEC)  training  occurs 
in  those  operational  settings. 

•  A  DMO  system  that  meets  the  needs  of  a  deployed  pilot  will 
fulfill  a  number  of  important  criteria  simultaneously.  It  will: 

—  Train  specific,  critical,  individual  job  skills, 

—  Provide  practice  on  individual,  intra-team  and  inter-team  skills,  and 

—  Provide  realistic  missions  in  adverse,  non-permissive  environments. 
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High  Fidelity  DMO  Environment 


Deployable  DMO 
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Deployable  DMO 
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Deployable  F-16  Cockpits 
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Deployable  F-16  Cockpit 


•'  " 

Mission  Training  Fidelity 
Requirements 


What  are  the  issues  in  building  a  deployable  DMO  system? 

—  Fidelity:  Designers  try  to  create  physical  (look  and  feel)  and 
functional  (dynamics)  fidelity  in  training  devices. 

•  Higher  fidelity  generally  leads  to  better  training  devices,  but 

•  Increases  in  fidelity  increase  cost,  so 

•  Tradeoffs  among  training  objectives,  costs  and  physical  and 
functional  fidelity  are  inevitable. 

—  Component/option  variations:  A  deployable  DMO  system  involves 
almost  limitless  potential  combinations  of  options  that  it  is  not 
possible  to  empirically  evaluate  all  possible  systems. 

•  FOV,  IGs,  software,  threat  models,  databases,  etc. 

•  Configurations  change  rapidly!  Where  to  spend  $$? 

—  Empirically  Assessing  Deployable  Systems:  Focus  on  systems 
that  prepare  warfighter  for  combat  experience. 
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How  to  Evaluate  Fidelity  and  its 
Effects  on  Utility? 

*  Need  applied  method/process 

—  Based  in  warfighter  needs 

—  Efficient 

—  Cost  effective 

—  Iterative 

—  Produces  data-driven  recommendations  for 
procurement  and  engineering  upgrades 

*  Goal  is  to  understand  (through  data)  the  training  utility 
by  mission  essential  area  and  to  provide 
priorities/recommendations  on  how  to  upgrade. 

*  Higher  level  findings  and  recommendations — i.e.,  not 
research  into  detailed  fidelity  levels  (e.g.,  JNDs) 
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Fidelity  Evaluation 
(Documentation?) 


Resolution 

1280x1024 

1280x1024 

1280x1024 

Scan  scheme 

Non-interlaced 

Non-interlaced 

Non-interlaced 

Type  of  display 

7ch  VI DS 

Torroid 

4ch  VI DS 

Horizontal  FOV 

360 

270 

216 

Vertical  FOV 

135 

70 

135 

Contrast 

30:1 

10:1 

10:1 

Sim  code 

Fortran/C++ 

Unk 

Ada/C/C++ 

G-suit  capable 

No 

Fittings  only 

No 

#  geogr.  areas 

1 

5 

1 

Blurring? 

Yes 

No 

Yes 
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Mission  Training  Fidelity 
Requirements 


Start  with  the  MECs 

—  An  Empirical  Method  of  Assessing  Deployable  Systems:  Focusing 
on  preparing  the  warfighter  for  combat  experience  led  us  to  use  of 
the  MEC  outputs.  The  MEC  process  identified  (across  A/A,  A/G,  & 
SEAD  missions)  171  F-16  experiences  crucial  to  combat  mission 
success,  including,  for  example: 

•  Performing  buddy  lasing 

•  Employing  weapons  with  degraded  systems 

•  Large  force  employment 

•  Seeing  surface-to-air  threat  launches  under  NVG 

•  Degraded  aircraft  avionics 

The  171  experiences  (plus  27  emergency  procedures)  are  the  core  of  a 
set  of  system  evaluation  criteria  to  be  used  by  SMEs.  The  198  evaluation 
points  organized  into  four  mission  areas  (A/A,  A/G,  SEAD,  EPs). 
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MEC-Based  System 
Evaluation  Survey 


We  are  trying  to  identify  the  extent  to  which  you  can  gain  various  experiences  with  the  simulator  (or  system)  you  have  just  become  familiar  with. 

Please  check  the  box  that  corresponds  to  the  extent  to  which  you  can  or  cannot  obtain  each  experience  in  this  simulator/system.  The  system  to  be  rated  here 
is:  DTT  verl  (see  specification  sheet  for  all  technologies/systems  included) 

To  what  extent  are  you  able  to 
gain  each  experience  below? 

Rating  Scale  to  be  used: 

0 

N/A.  (Can’t 
experience  at 
all — capability 
does  not  exist) 

1 

Capability  to 
experience 
exists  to  a 

very  poor 

extent 

2 

Capability  to 
experience 
exists  to  a 

poor  extent 

3 

Capability  to 
experience 
exists  to  a 

moderate 

extent 

4 

Capability  to 
experience 
exists  to  a 

qood  extent 

5 

Capability  to 
experience 
exists  to  a 

verv  qood 

extent 

i 

Employing  a  Maverick 

2 

Strafing 

3 

IR  fog 

4 

Participating  in  coalition/joint  strike  package 

5 

Unlocated  or  pop-up  threats 

6 

...(survey  continues  with  all  experiences  listed 
as  rows)... 

Note:  All  experiences  (rows)  taken  directly  from  (a)  the  results  of  the  MEC  process  for  that  domain  &  mission,  and 
(b)  dash  one  bold  face  emergencies  (e.g.,  flameout  landing).  ^ 


Research,  Evaluation,  & 
Improvement  Flow  Chart 


Identify  platform 
&  mission(s) 


Document 

system 

f Overall  MEC 
^  Exp  eval 
survey  w  dash 
one 

Step  to  sim 
(or  system) 

Devel- 


1Weightings  on  the  experiences/EPs  may  be  needed  to  more  closely  align  with 
intended  training  priorities/needs 

2  Major  decision  point,  as  this  will  determine  if  sim/system  is  up  to  standards.  ((  DONE 
Comparisons  between  systems  (i.e.,  fidelity)  could  be  made  here  &  associated 
with  actual  performance  benchmark  data  to  ascertain  extent  of  fidelity  trade-offs. 


— ►  3Determine  deficiencies 

-O_ 3 

Analyze/ 

report 

v  f  MEC  exp  X 
deficiency  survey  / 

Determine 

solutions/ 

costs 


I 

Procure/ 

Integrate/ 

Test 

T 


etum  to 
box  #2 


3  Deficiencies  should  be  listed  and  prioritized 
at  the  most  elemental  functional  level, 
f  Example  surveys  for  A  and  B  shown  next 
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Preliminary*  Baseline  Results 


*  Survey  administered  for  both  deployable  system  and 
high  fidelity  system 

*  Deployable  system  rated  highest  for  A/A  and  A/G 
mission  areas 

—  System  rated  very  low  for  SEAD  and  EPs 

—  System  designed  with  A/A  and  A/G  in  mind 

*  High  fidelity  system  showed  similar  trends,  but  over  50* 
experiences  rated  significantly  higher. 

*  Next  step  is  to  analyze  lowest  rated  items  for  common 
underlying  technological  deficiencies  (e.g.,  visual  field, 
IOS). 


Then... 
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Deficiency  Evaluation  Survey 


i 


We  are  trying  to  identify  the  extent  to  which  different  categories  of  DMO  deficiencies  detract  from  the  ability  to  gain  the  MEC  experiences  &  practice  dash 
one  bold  face  emergencies  (e.g.,  in-flight  flame-out).  Please  rate  to  what  extent  each  category  (in  column  headings)  degrades  your  ability  to  gain  each 
experience  (rows)  for  the  identified  DMO  system.  The  system  to  be  rated  here  is:  DTT  verl. 

0  =  Not  at  all;  1  =  To  a  Slight  Extent;  2  =  To  a  Moderate  Extent  3  =  To  a  Substantial  Extent;  4  =  To  a  Great  Extent 

How  much  is  your  ability  to  gain  experiences... 

DMO  subsystem  deficient  areas  to  be  rated 

...  under  these 
conditions... 

degraded  by 
these  current 
configurations? 

r" 

Current 
visual  field 
(i.e.,  degrees 
of  visual) 

Current 
level  of 
visual 
acuity 

Current 
level  of 
jamming 
(all  forms) 

Current 
brief/debri 
ef  system 

Current 

threat 

generation 

system 

Current 

lack 

IOS 

fault 

insert. 

1 

Employing  a  Maverick 

2 

Strafing 

3 

IR  fog 

4 

Participating  in  coalition/joint  strike  package 

5 

Unlocated  or  pop-up  threats 

6 

...(survey  continues  with  all  experiences  listed  as  rows)... 

72  A/G,  55  SEAD,  &  45  A/A  experiences  (172  total) 

Note:  All  experiences  (rows)  taken  directly  from  (a)  the  results  of  the  MEC  process  for  that  domain  &  mission,  and 
(b)  dash  one  bold  face  emergencies  (e.g.,  flameout  landing). 


How  process  and  data 
could  be  used 


*  Well  suited  for  evaluating  tactical-level  tasks  (i.e.,  the  experiences). 

*  Could  easily  be  used  immediately  on  any  existing  sim/system  that  has  the 
MECs  defined  for  that  domain/mission. 

*  Could  be  used  to  certify  sims/systems  on  tactical-level  training  tasks. 

*  Could  easily  compare  impact  of  a  single,  controlled  technology  change  on 
training  the  experiences  (e.g.,  fly  sims  with  and  w/o  20/20  visuals  to 
determine  added  value  on  entire  cross-section  of  “real-world”  tasks). 

*  Could  compare  across  sites  and  fidelity  levels  (especially  when 
augmented  with  standard  assessments  such  as  benchmarks) 

*  Identifies  and  prioritizes  deficiencies  and  potential  upgrades — use  raw 
data  or  augment  with  MEC  gap  analysis  work. 
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Questions? 
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Mission  Training  Fidelity 
Requirements 


•  Once  a  deployable  DMO  system  is  available  for  training 
applications,  a  number  of  research  issues  arise.  Research  will 
need  to  address: 

—  Skill  acquisition  rates:  Which  MEC-based  experiences  can  be  trained  in 
the  deployable  DMO?  How  quickly? 

—  Skill  decay  rates:  Once  acquired,  how  rapidly  do  these  skills  decay  in  the 
absence  of  CT?  What  is  the  optimal  schedule  of  CT? 

—  Skill  transfer  attributes:  How  extensive  is  the  transfer  of  MEC-based  skills 
from  deployable  DMO  training  to  live-fly  training  and  to  combat?  What 
elements  of  proficiency  must  be  trained  in  live-fly? 

•  To  address  these  questions  a  performance  measurement 
approach  must  be  developed  that  quantitatively  assesses 
proficiency  in  a  standard,  controlled  way:  the  Standard 
Assessment  Set  (SAS). 
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Summary 


*  DMO  proven  effective  training  solution,  but  they  have 
not  migrated  to  the  field 

*  How  to  determine  technologies/fidelity  to  integrate  into 
these  environments? 

*  Propose  new  method 

—  Mission  Essential  Competency  based 

—  Understand  system  strengths  &  weaknesses  in 
terms  of  warfighter  needs 

—  Understand  technology/fidelity  limitations  and  their 
pervasiveness  across  the  warfighter  needs. 

—  Prioritize  upgrades 
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